Summary

JobManager.GetRobotsRules previously refetched /robots.txt on every call.
The stream worker's job-info cache expires at five-minute intervals, so a
long crawl re-hit the origin every five minutes — and a 429 on
/robots.txt came back on the next read with no negative caching. On
2026-05-11 the merrypeople.com run produced two 429s on /robots.txt
five minutes apart. No functional damage (the code falls through to "no
restrictions") but unnecessary origin load and noisy logs.

This change adds a per-domain in-memory cache with:

• Positive TTL: 1 hour. A long crawl now fetches /robots.txt once
  per hour at most.
• Negative TTL: 60 seconds. A transient 429 on /robots.txt
  unblocks within a minute and stops re-hammering the origin in the
  meantime.
• singleflight collapse: concurrent misses for the same domain land
  on one origin fetch instead of N parallel requests.
• Domain key: lower(util.NormaliseDomain(domain)) so case-variant
  and scheme-prefixed inputs share one entry.

Restart loses the cache, which is fine — a single fetch on the next
read warms it. No Redis involvement (single worker box; in-memory is
the right granularity).

Test plan

• go test -race -short ./... passes
• scripts/security-check.sh clean
• Tests cover:
  • hit-then-cache positive TTL (10 reads → 1 origin fetch)
  • error caching for negative TTL (5 reads of a 429 → 1 origin fetch)
  • positive TTL expiry triggers refetch
  • negative TTL expiry triggers refetch (and can recover to success)
  • case/scheme normalisation collapses onto one entry
  • 20-way concurrent miss → 1 origin fetch via singleflight

Risk

• In-memory only — restart loses the cache. Acceptable: cold start
  triggers at most one fetch per domain seen during the next crawl
  window.
• The cache lives on JobManager, which is a long-lived singleton in
  the worker process — no leak risk because the key space is bounded by
  the number of domains the worker has ever touched (≪ memory budget).

^{Need help on this PR? Tag @codesmith with what you need.}

• Let Codesmith autofix CI failures and bot reviews

Summary by CodeRabbit

• Changed
  • Robots.txt and sitemap discovery now caches results per canonical domain (1h positive, 60s negative TTLs), collapses concurrent fetches and bounds refetching to reduce redundant origin requests and rate-limit responses.
• Tests
  • Expanded coverage for positive/negative caching, TTL expiry, concurrent request collapsing, cancellation handling, and domain canonicalization.